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ABSTR ACT 



A research experiment was conducted to investigate how 
accurate a voice recognition system must be for daily 

production use. Specifically, the purpose of the research 
was to establish the percentage accuracy level at which a 
user becomes frustrated and decides not to use a voice 
recognition device. The experiment consisted of controlling 
the perceived recognition accuracy of a voice recognition 
system and then collecting data through the use of a 

guestionnaire from the experimental users on th*^ 
acceptability of the equipment. The experiment was not 
totally successful for a variety of reasons. This paper 
will discuss the research methodology, review the data 
collected, and suggest possible alternatives to the 

design to overcome 
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User Accuracy Requirements 
For Voice Recognition 



I. INTRODUCTION 

Within the last ten years voice recognition technology 
has grown from a laboratory research endeavor to a useful 
and economic, computer human interface tccl. The equipment 
available today is relatively inexpensive, compact, and 
accurate as evidenced by numerous applications, both 
industrial and military, which successfully use this input 
methodology. Improvements in the technology are still being 
made. Three major areas for continued research in voice 
input to computers are as follows. 

1. User independence 

2. Continuous speech 

3. Vocabulary capability of 10,003 to 12,000 words 

The literature indicates that the solu''-ion or at least 
incremental breakthroughs to these three research areas are 
just a few years away. Hopefully, the goal of manufacturers 
and researchers is to develop and produce systems with all 
or some of the properties listed above which have the best 
accuracy rate possible. A question arises with this goal in 
mind. At what accuracy rate should the system be made 
available to users? If an accuracy rate of 95% is 
acceptable why not use those systems while new and better 
algorithms are being developed? Conversly, if 95% accuracy 
rate is net acceptable and will give voice recognition a bad 
reputation among management personnel and users it should be 
held back until an acceptable rate can be achieved. A 
measure of where this accuracy threshold is could be of 



1 



great use to researchers and system manufacturers. 
Therefore, the question investigated was how poor must a 
voice recognition system be before the user becomes 
frustrated with the error rate and will choose not to use 
the system. 

A required accuracy rate will depend on the task for 
which the system is intended to be used. For example, if 
voice recognition is to be used for the input of guidance 
parameters and launch sequence commands in a computerized 
missile delivery system, anything less than 99+% accuracy 
would be unacceptable even under the high stress situations 
most likely existing during the input process. On the other 
hand, there are numerous tasks which can be labeled as non- 
critical where voice recognition errors can be tolerated and 
in fact now occur frequently using more conventional input 
methods such as a keyboard. A typical task in this category 
would be information retrieval such as a stock broker 
obtaining stock information for a client, or an airline 
reserva tionist retrieving flight information for a customer. 
Neither of these tasks are particularly critical in nature. 
If an occasional error is made during the input process it 
is easily corrected and the process repeated without any 
damage to the system or loss in revenue, profit or system 
efficiency. 

An experiment was developed to study the user 
accepta biltiy question with the objective of determining a 
user required accuracy rate. A VET/2 voice recognition unit 
manufactured by Scott Instrument Inc. of Denton, Texas was 
used in conjunction with a Basic software program created on 
an Apple II Plus microcomputer. The experiment's subjects 
were asked to read an ordered list of words into the voice 
recognition unit and observe whether the word was recognized 
properly. This same ordered list of words was stored in 
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computer memory. Each subject was led to believe the system 
was in a recognition mode, but in fact, the software program 
waited for a verbal input and then drew a random number to 
determine if the spoken input should be displayed as correct 
recognition, misrecogni tion or nonrecognition. If the 

random number determined it was to be a correct recognition 
the next word on the ordered list was displayed. If a 
misrecognition was to occur an alternate word was displayed, 
and a question mark was displayed if the random number 
determined a nonrecognition was to appear. In other words, 
the program can be viewed as a voice actuated system where 
any verbal response would trigger the recognizer, but a 
random number would determine the output, not the 

recognition algorithms. As long as the subjects continued 
to read the ordered list of words, it appeared as if word 
recognition was being accomplished. By varying the random 
number test the subject's perceived accuracy rate could be 
controlled. The errors were recorded, and each subject was 
asked to complete a questionnaire. The questions were 
designed to indicate when a frustration level was reached 
due to recognition inaccuracy. 

The experiment did not lead to the desired goals or 

results. This paper will cover the experiment and why -^.he 
actual results were different than predicted. First, ■'rhe 
report will describe the computer software program which was 
created to vary the perceived recognition rate during the 

experiment. Next, the experimental design will be 

discussed. The entire design was not implemented because 
preliminary data analysis ir.dica-^ed the desired results were 
not being obtained. The method used to implement the 

experimental design will also be discussed. The ■^hird 
section of the report will cover the preliminary data 
analysis and summary of user responses to the experiment. 
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Finally, conclusions will be drawn as to why ■’■he experiment 
did not resolve the question at hand, and recommendations 
will be presented for future research concerning the 
question of "How good should voice recognition equipment 
be?". 
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II. PROGRAM SOFTWARE DESCRIPTION 



Two computer software programs were written for use in 
this experiment. The first program was used to create three 
databases consisting of words to be used for the planned 
experimental design. Since this program was used only as an 
aid for the database preparation, it will not be discussed. 
A copy of the program and the databases of words are 
attached in Appendix A and Appendix B respectively. The 
second program created was used to alter the perceived 
recognition accuracy of the Scott Instrument VET/2 

recognizer. It was written in Applesoft Basic and is 

included as Appendix C. The following description explains 
the program logic using the program line numbers for 

ref eren ce. 

Line 10 dimensions a character array called WS (200,2). 
During the experiment, this array held the 200 vocabulary 
words used for the experiment. W$(I,1) held the word which 
was to be spoken, and W$(I,2) held a sound alike word. The 

use of this sound alike word will be explained later. The 

array F$(3) held the name of the three word databases which 

were available for use. These databases were named: 

1. COMPUTER, designed to be used by users of a 

text editor on the IBM 3033 computer at the Naval 
Postgraduate School, 

2. STOCK, designed to be used by stockbrokers, and 

3. AIRLINE, designed to be used by airline 
reservationists or travel agents. 

The cirray P(4) held the four different probabilities 
associared with the planned recognition rates of 99%, 95%, 

90%, and 85%. 
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Line 20 sets the variable D$ to a control D which is used 
for file manipulation on the Apple computer. 

Lines 30 and 40 are data assignment statements for the F$ 
and P arrays. 

Line 45 assigns to the character variable M$ a series of 
five blank spaces. M$ is used for prin^. control or print 
spacing . 

Line 50 is an output statement asking the user what 
database is to be used. 

Line 60 accepts as an input the number of the desired 
database. This number is placed in the variable A. 

Line 70 opens the correct database file. 

Line 80 sets the read device to the appropriate database 
file. 

Line 90 reads in the 200 word vocabulary and their sound 
alikes into the W$ array. 

Line 100 closes the input database file. 

Line 105 asks for a random number generator seed and 
places the integer response into the variable called IS. 

Line 110 asks for an algorithm number, which in effect is 
the array index of the desired accuracy rate. The question 
in line 110 was S'*:ated in such a manner so tha+ experiment 
participants would assume different word recognition 
algorithms were being tested. 

line 120 places the algorithm number in the variable B, 
and if B is less than zero, the program is stopped. 

Lines 130 through 160 call the Scott instrument voice 
recognition subroutines used to load the voice patterns into 
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The voice 



memory, and initialize the recognition unit, 
patterns, although not used for recognition purposes, were 
neccessary for the proper operation of the recognition unit. 

Line 170 prints a header announcing the practice session 
of 10 words. During this practice session the use of voice 
recognition eguipment was explained to the experimer.-^ 
participant. The explanation given to each participant will 
be described in detail later. 

Lines 180 through 200 create a program loop. The 
variable I is used as an indexing variable. This indexing 
variable is first set to 1, and the first word of the 
vocabulary database is printed on the Apple computer- 

display. This display is used .as a prompt to *he 

experiment's participants for the word they are to speak. 
After the word is displayed the program is transferred to 
the recognizer's subroutine which will accept a verbal 
input. After the recognizer accepts the voice input the 
subroutine returns control to this main program. After the 
acceptance of the voice response an artificial delay is 
created by the "FOR Z ..." statement. This delay was 

neccessary to provide a capability of stopping program 
execution if the participant made an error which could lead 
them to believe the program was not actually recognizing 
their voice. The delay provided the time for -^he 

experimenter to stop the program before the "recognized" 
word was displayed. After the delay a subroutine, which 
will be described later, is called to determine whether a 
correct response, a sound alike mistake, a random mistake, 
or a nonrecognition response should be displayed to the 
participant. After the response is displayed the index 
variable, I, is checked to determine if the test practice 
session is over. 
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line 210 sets a series of counters to zero. These 
counters keep track of the number of non recognitions 
presented to the participant (Q), the number of sound alike- 
misrecognitions presented (S) , and the number of nonsense 
misrecognition responses (N) . The variable T is calculated 
at the end of each particpant* s pass through the 200 word 
vocabulary list and holds the accuracy rate actually 
presented to the subject. 

Lines 220 through 250 comprise another program loop. The 
logic is similar to lines 180 through 230 except the entire 
200 word vocabulary is seguentially displayed. 

lines 260, 270 and 280 respectively display statements 

thanking the participants, calculating the actual accuracy 
rate presented, and displaying the accuracy rate and all 
counters in a coded form. 

line 290 sends control of the program back to the 
guestion asking which algorithm should be used (Line 110), 
and the program is ready for the next participant. 

line 300 is the first line of the subroutine which will 
calculate whether the response which is to be shown to the 
participant is a correct recognition response or one of the 
three possible error responses. A random number is drawn. 
If the random number is greater than the accuracy rate which 
is presently being simulated the program will branch to the 
statements neccessary to calculate the type of error which 
should be presented. 

Line 310 is executed if a correctly recognized response 
is to be displayed. The print statement will first prin^ 
the variable containing the blank spaces, and then the 
correctly recognized word. This was done so the recognized 
word was displayed further to the right on the Apple screen 
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than the word which was output as a prompt. Transfer then 
is passed to statement 400. 

Lines 320 trhough 340 are used to determine the type of 
error that should be displayed given that the present word 
is to be perceived as an error by the participant. A random 
number is drawn, and if it is less than .33 the error is 
considered a sound alike error and control is passed to line 
390. If the random number, R, is between .33 and .56, it is 
considered a random or nonsense error and con-*-rol is passed 
to line 360. Finally, if the random number is greater than 
.66, it is considered a nonrecognition. Therefore, th<= 
three types of errors are equally likely. In previous voice 
recognition studies, Poock ( 1980) , Poock(1981), and 
Jay (1981) , error rates of about 1.8% have beer, consists n'^ ly 
experienced with a Threshold Technology Inc. , Delran New 
Jersy, model 600 voice recognizer. Ir these studies non- 
recognitions consisted of 31% to 35% of the total recorded 
errors. Ther<^ were no statistics available on the 
percentage of misrecogn itions which could be considered 
sound alikes or non-int uitive confusing misrecognitions. 
For this reason it was assumed that misrecognitions should 
be equally divided between the sound alike possibilities and 
the random error possibilities. Therefore, all three error 
types were programmed to occur with equally likely 
protabilities. 

Line 350 is executed if a non recognition is to occur. 
It again prints the variable, M$, which contains blanks and 
then a question mark, ?, representing the Scott Instrument 
convention for a non-recognition. The counter for non- 
recognitions is increased by one and the program is then 
transferred to line 400. 

Line 360 through 380 determine a random word r'^sponse for 
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the random misrecog nit ion case. A random number is drawn 
and converted into a random integer between 1 and 200. N^xt 
it is checked to ascertain that the random integer generated 
is not the word which is to be misrecogni zed. If this check 
had not been made, a misrecogni tion could have been recorded 
but the participant would have in fact seen the correct 
response. The randomly selected word is printed on the 
display in the same manner as a properly recognized word, 
and the nonsense word counter is incremented. Again control 
is then passed to statement 400 . 

Line 390 is executed if the random number, R, indicates 
that this incorrect recognition should be a sound alike and 
the second word in the WJ array is printed out. These words 
have been selected in such away that an average user would 
conclude that it was an understandable or likely recognition 
error. The sound alike counter is incremented. 

Line 400 is the last line of the subroutine. Again a loop 
is added to produce a delay. The index used in the program 
loop consisting of statensn-^s 220 through 260 is 
incremented. A blank line is displayed for readablity and 
the subroutine returns to the main program to print the next 
word in the 200 word vocabulary list. 

The next section will describe how this program was used, 
and the reaction of the experiment's participants as to the 
believa bility that a voice recognizer was being tested. 
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III. EXPERIMENTAL DESIGN 



It has already been mentioned that ■^hree databases w=>re 
formed for the research experiment. The plan was to run 
three groups of 100 people each through one of the 
databases. In other words, 100 stockbrokers or investment 
counselors would say the 200 words associated with their 
profession, and then rate -^he acce ptability of th<= 
equipment. Likewise, 100 airline reservationists or travel 
agents were to use the airline reservationist word list, and 
100 students would use the 200 words associated with the IBM 
3033 text editor program. 

A word list of 200 words was used for two reasons. First 
it was decided that going throuah a list of 100 words went 
too fast, and the subjects would not get a good feeling for 
the accuracy rate. Using 300 words was definitely out of 
the question because of the time involved in conducting the 
experiment for the number of planned subjects, and because 
of a possible boredom factor which could complicate the 
subjects perception of the system. The median seemed like a 
reasonable choice. The vocabulary size of 200 words also 
satisfied a second criteria. That was the desire to get 
user frustration information at a more accurate level than 
every 1 percentage point. At least the 200 words would give 
ratings at every one half percentage point. 

During the experiment it became apparent that a 
frustration level was not being achieved or at least 
measured. Therefore, after the first fourth of the 

experiment some preliminary data analysis was done, and •'rhe 



re suits 


showed 


that 


little was 


being learned about 


user 


accuracy needs. 


T t 


was decided 


to stop the experimen-^ 


and 


re pert 


on what 


had 


been dona 


to date. A total of 


78 
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subjects participated in the test. The intent of this 
section of the paper is to explain how the subjects were 
introduced to the experiment and how they reached to the 
recognition system. This section will also discuss the 
design of the questionnaire. 

The subjects were students, staff and faculty members at 
The Naval Pcs-^ graduate School in Monterey, California. They 
were all volunteers and between the ages of 25 and 49. The 
entire explanation and experiment took between 10 and 15 
minutes per subject. 

When a subject arrived at the experiment site, it was 
explained that some new user independent- voice recognition 
algorithms were being tested. The Scott Instrument voice 
recognition device was covered to preclude the subjects from 
getting the wrong impression of its capability. The 
algorithm number was entered into the system in front of the 
subject as the idea of different algorithms was being 
explained . 

Some of The subjects had used voice recognition equipmen-*-. 
on previous voice experiments. These students needed little 
practice, but still went through the ten practice words. 
The practice words were used as the teaching device for 
those subjects who had not used the equipment before. The 
need to speak a phrase as a continuous flow of speech was 
explained, as was the explanation of the ra«^aning of a 
question mark (?) when it appeared on the display. If the 
subject showed an interest in the machine's capability 
during the '■he practice session, their questions were 

postponed until after they had answered '•he questionnaire. 

The subjec'-s were asked to ignore all the errors (if any) 
which occured during their practice session. It was 
explained that some artificial intelligence algorithms were 
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being employed and the system was selec*:inq characteristics 
of their voice for use in the main portion of the 
experiment. 

As the subjects were going through the 200 word lisr, the 
experiment was stopped each time an error occured. The 
error was pointed out to the subject and a short explanation 
was given as to what that error would do if the true tex+ 
editor was employed. The idea for stopping •'■he program was 
two fold. First it was noticed during preliminary program 
testing that some people started to read the words on the 
display and weren't watching the recognized word display<=d. 
In ether words, the experiment bored ■'■hem and they weren't 
always aware of ■'•he errors. At first this had been solved 

by placing the recognized word directly under ■^he prompt 

word. Unfortunately this solution caused another problem. 
The subjects who participated in the preliminary testing 
started to say the recognized word instead of the following 
prompt word which had a devastating effect on the user's 
confidence that the system was recognizing ■^heir voice. 

Therefore, the recognized word was placed to the right of 

the prompt word, and each error was pointed out to the 
subject. The second reason for stopping ■^.he program was to 
delay the subject in completing the experiment. It was 
hoped that the idea of an error slowing them down would 

transfer to their perception of how the system would work in 
a real environment. The plan was to slow them down thus 
creating a frustration level which was to be measured. 

There were numerous times even with the precautions taken 
where the subject read the wrong word, or started to make a 
comment without the microphone being turned off. This led 
to a recognition which in the majority of cases was correct, 
when the spoken utterance was obviously incorrect. The 
experimenter's finger was always kept lightly on the Apple 
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computer’s space bar. By depressing the bar the microphone 
and voice recognition system were deactivated. The majority 
of the time, the experimenter depressed the space bar soon 
enough to avoid a correct recognition of an incorrect voice 
input. This was the major reason for the delay loop 
explained earlier in the program software description. If 
the space bar was not depressed soon enough, and the correct 
response appeared on the display, the subject was told that 
the software program had been developed to do its own data 
collection automatically. Furthermore, it was explained 
that the experiment was interested in only recognizer errors 
not the human errors which will always occur with a voice 
system. Therefore, it was explained, that pressing the 
space bar was an automatic override, and no matter what was 
said the automatic data collection routine would count it as 
a correct recognition. This explanation seemed to satisfy 
everyone, who encountered the situation. 

After the subject had completed the 200 words, he or she 
was asked to fill out a two page questionnaire. This 
questionnaire is attached as Appendix D. Questions 3,4 and 
5 created the data which was of most impor'*-ance to the 
experiment and was the measurement of user acceptability. 
The sets of response alternatives, for these three 
questions, were taken from an Army Research Institute 
publication on questionnaire construction (1976). The 
responses have been tested and shown to have mean scaling 
factors at least one standard deviation away from each other 
while maintaining the parallel wording. Question 9, 

concerning the part of the country the subject grew up in, 
had no‘*:hing to do with the experiment that was being 
conducted, but was included to make ^he experiment about 
testing user independent algorithms more believable to the 
participants. The remaining questions ace self explanatory. 
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and will be reviewed in detail when discussing the analysis 
of the data collected. 

After the questionnaire was completed the subject was 
free to ask questions about the system. They were led to 
believe that all algorithms had been created at the School, 
and that the technology was not commercially ready because 
of the extremely large amount of core needed to run the 
system. This was done to insure that the believability in 
the system would not decrease as studenhs talked about the 
experiment. On ■'■he other hand, it was not the in^'-ent of the 
experiment to lead the subjects to believe that voice 
recognition capabilities were beyond the present state of 
the art. 

The experiment was a total success as far as the 
believability of the system was concerned. There w=^re a 
couple of instances when the random number generator 

cooperated fully. For example, a subject asked a question 
which triggered a response and the random number generator 
created a nonrecognition. In another case a word was 

mi srecoqn ized in the test sequence and the same word was 
mi srecogn ized during the experiment, both times the 

misrecoqnition was the sound alike word. There were 

subjects who tried to analyze the system and hypothesized 
why the system did not recognize them correctly. One 

subject was convinced that any word with an ”S" sound would 
not be recognized properly because he tended to slur his "S" 
sound. There were only two subjects out of the 78 tested 
who mentioned the fact that they doub'-ed ^:he system was 
recognizing their voice. This fact was noted on th<=^ir 
questionnaire after they left the laboratory area. 
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IV, DATA ANALYSIS 



Appendix E and F present the data collected. Appendix E 
contains the raw data collected, while Appendix F has the 
data in the ranked form. All of the data analysis used 
nonpara metric statistics methodologies based on ranks. 

The first column of data in Appendix E contains the total 
number of errors the user observed. This number is the sum 
of nonrecognitions and both types of misrecognitions. The 
second column contains the total number of misrecognitions 
which is the sum of the sound alike errors and nonsense 
errors which were observed by the subject. Columns 3,4, and 
5 are the individual error totals for nonrecogr.i-^ ions , sound 
alike misrecognnitions, and random misrecognitions, 
respectively. Column 6 is the age of the subject, and 
column 7 is the average number of hours the subject spends 
using a computer terminal each day. The average number of 
hours spent at a terminal were considered important for the 
stock brokers and airline rese rvat ionists, but had little 
meaning for the students who became the only participants in 
the experiment. Therefore, this data will no-*- be used in 
the analysis, but is presented for completeness. 

The next two columns, column 8 and 9, are the answers to 
the questions on whether the subject had ever seen or used 
voice recognition eguipment. These answers are coded with a 
1 representing an answer of "yes", and a 2 representing a 
"no" response. Columns 10, 11, and 12 are the responses to 
the guestions dealing with user acceptability. These are 
also coded, from a 1 meanina a poor acceptability response 
to 5 for a high acceptability response. Column 13 is the 
sum of the responses to the three questions. Since the 
questions each had parallel wording the sum of the answers 
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for the individual questions was used for the data analysis. 
This method gave a more accurate numeration of the 
acceptability level for each subject. 

Column 14 and 15 are the data collected on the subject's 
perception of his or her own typing ability in terms of 
speed and accuracy respectively. These data fields are also 
numeric codes for the response given on the questionnaire. 
A value of 1 represents a poor rating, a 2 an average rating 
while a 3 represents a very good rating in the speed and 
accuracy capability of each subject. Column 16 is the 
response to the question of whether the subject ever had a 
typing course. The same response convention was used for 
the two previous yes-no questions. 

The second to the last column, column 17, tabulates the 
data collected representing the geographic region where the 
subject grew up. The codes have the following meaning. 

1 - South 

2 - East 

3 - Midwest 

4 - Foreign 

5 - West 

6 - All over or not specified 

Finally the last column, column 18, is the subject's 
number of years experience with computers and computer 
termina Is . 

As it was already mentioned Appendix F contains the 
ranked data. The data are organized in the same manner as 
previously described for the raw data except some of the 
columns are the ranks of the data collac‘:ed. There were 
numerous ties and the rank value assigned was the average of 
the ranks that would have been assigned to them had there 
been no ties. All of the recognition error counts were 
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ranked, as were the age data, total acceptability rating 
data, and the subject experience data. 

The first set of analysis used Spearman's rank 

correlation coefficient (Conover, 1980) as a test statistic 
to determine if there was any negative correlation between 
the number of errors presented to a subject and that 
subject's acceptability rating. It had been hypothesized 
that as the number of errors increased the user 

acceptability would decrease. Therefore, the null 

hypothesis was that the number of errors and user 

acceptability are mutually independent or had no 

correlation. The alternative hypothesis was that these two 
variables are negatively correlated. The following 

tabulates Spearman's correlation coefficient calculated on 
the ranked data between these two variables. 



all errors 


-.250 


tota 1 


misr ecogni tions 


-.249 


non r 


ecogn itions 


-.250 


sound 


alikes 


-.215 


random misrecogn itions 


-.201 



At a significance level of .02 5 rhe null hypothesis can be 
rejected for the first three values. It appears as if the 
desired negative correlation is present, but the correlation 
is very slight as indicated by the value of the correlation 
coefficient. The hypothesis of mutual independence can not 
be rejected between the sound alike errors and user 
acceptability, and the random errors and the acceptability 
var iabl e. 

Figure 1 is a graph of the numeric user acceptability 
totals versus total user perceived errors. It shows that 
although there is evidence of a negative correlation there 
is little information existing as to where or at what error 
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level the acceptability values start to decrease. In fact 
the real problem is exemplified by observing -^he average 
response values for various groups of subjects. Table 1 
contains, for various groups of subjects, the averages of 
the total values for the three accep-*:abil i-^y ques-t-ions along 
with the standard deviation. There is very little 
difference between the groups. Even the group that observed 
more than a 15% error rate still rat<=d the system in the 
"like it" and "would use it" category. A Kr uskal-Wall is test 
was done to determine if these groups of subjects had the 
identical mean response values. This hypothesis could not 
be rejected at the .05 or .1 significance level. The test 

statistic value, T, was calculated at 6.78 and the chi- 
square distribution quantile for the four degrees of freedom 
at the .05 level is 9.488 and at the .1 level is 7.779. 
Therefore, even though a small negative correlation is 
detected in the data, very little information can be gained 
as to where a distinct drop occurs in user acceptability 
va 1 ues. 

Two other Spearman's Rho correlation coefficient tests 
were done. First a correlation possibility was investigated 
between the ranked values of age and the ranked values of 
acceptability. The hypothesis that age and acceptability 
values were not correlated could not be rejected. 
Spearman's correlation coefficient was calculated at .03. 
The same test was done to check the data for mutual 
independence among the ranked values for years of experience 
and the ranked values of the acceptability totals. Again 
the independence hypothesis could not be reiected with a 
correlation coefficient of -.02. 

In addition to the Krus kal-Wallis test previously 
mentioned, a series of similar tests were completed in order 
to determine if there wer“= any differences in the mean 
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acceptability responses among different groups of 

individuals. Table 2 summarizes these tests. None of the 

hypotheses that all the mean acceptability responses were 
idenitical could be rejected. In other words no statisical 
differences could be found among the various groups tested. 

The only test which suggested a possible difference was 
the test between the groups divided by geography. This was 
interesting because numerous people approached the 

experiment asking for example, if this machine understood 
"Louisianian". They knew they had a distinct southern 
accent, and if it recognized them they were very surprised. 
This could account for the relatively high Kruskal- Wallis 
statistic even though the groups could not statistically be 
shown to have different means. 

In conclusion, there is really very little information 
present in the data collected. For this reason the 

experiment was cut short before the time was spent at 
business establishments. It is hypothesized that there are 
at least two basic flaws in the experimental design. First 
the experiment was working with an advanced technology. 
Some of the people who were tested were not aware that voice 
recognition existed. In fact, numerous people answered on 
the questionnaire that they had seen voice recognition to 
computers on "Star-Trek". With an at^i^-ude like that any 
recognition capability was impressive. If this is true it 

would be expected that there might have been a larger 
difference between the group acceptability averages between 
those who had seen voice recognition ejuipment before and 
those who had not. Since this did not occur we can only 
assume that the group which had seen voice recognition 
before knew about its user dependence limitations and were 
equally impressed with the user independence capabilities 
being demonstrated. 
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The second problem area involves the lack of a task which 
needed to be accomplished within the experimental framework. 
Reading a list of words and pointing out the errors did not 
create the frustrating situations which are going to exist 
when you encounter a recognition error while trying to 
accomplish a task. Although it was hoped that stopping the 
experiment each time an error occurred would provide this 
feeling, it did not totally simulate the frustration 
associated with task completion. Furthermore, subjects who 
had never seen or used voice recognition equipment had a 
difficult time visualizing how the equipment would actually 
be used. Although an explanation was given to each subject 
at the beginning of the experiment, it appears as if the 
concept was not totally understood by everyone. This was 
evidenced by some of the questions asked by the subjects 
after the experiment was finished, 
realistic job or goal, this problem 



If there had been a 
could be alleviated. 
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5-0 7.5 JO.O 12.S J5.0 17.5 20.0 22.S 

PERCENTAGE ERROR RATE OBSERVED 



% Error Rate 
Group s 


# of sub j ect s 
in group 


Average 


Standard 

Deviation 


0.0 - 1.5 


19 


4 .40 


4.46 


2.0 - 5.0 


13 


4.41 


4.49 1 


5.5 - 10.0 


23 


3. 96 


j 

4.08 j 


10.5 - 14.5 


16 


3.71 


3.93 * 

♦ 


15.0 - 21.0 


7 




3 . 90 

1 


4.02 1 

\ 

1 



TABLE 1. 

Acceptability Response Summary 
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1 

GROUP 


TEST STATISTIC 


CHI-SQUARE = .05 


Seen Voice equipment- 
not seen voice equipment 


1.18 


3. 84 

j 


Used Voice equipment- 
not used voice equipment 


3 . 65 


1 

1 

3.84 j 


Slow, intermediate, fast 
speed typist 


2.14 


] 

5.99 i 

f 


Poor, fair, very good 
typing accuracy 


4 .38 


J 

5.99 1 


Typing course- 
no typing course i 


0.12 


< • 

i 3.84 1 

.... ! 


Part of country raised ; 

i n 


; 10.15 


1 

11.07 i 

t 



TABLE 2. 

Results of Krus ka 1 - Wa 1 1 i s Tests 
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V. RECOMMENDATIONS AND CONCLUSIONS 



The data collected in this experiment do not help answer 
the question of how good a voice recognition system must be 
to maintain user acceptability. The following 
recommendations are provided for future research efforts and 
experiments. 

1. Introduce the subjects to voice recognition- 
equipment before the start of the experiment. 

2. Demonstrate the equipment wi^-hin an appropriate 
work environment. In other words, demonstrate •'■he 
use of the equipment to accomplish a suitable job 
within their individual areas of expertise. 

3. Have the subjects train the equipment. Don'-^ 
present a system which is obviously years ahead of 
currently available technology. 

4. Set up a series of tasks which are suitable for 
voice recognition input which can be accomplished 
within the test environment. It appears that it 
is important that the job is very realistic in 
nature. 

5. Consider the following method to vary the 
recognition accuracy for the experiment. Use a 
recognition system which has the capability to 
easily access not only the word which was 
recognized, but the runner up word. Then by the 
use of a random process determine whether '^he 
recognition unit should output the first or runn<=r 
up word. If the recognition system used has a 
fairly good accuracy rate, the the first choice 
word should be the correct word. Therefore, by 
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randomly selecting the second word you are 

randomly selecting errors from the recognition 
unit. It is possible that the recognition unit 
will make a mistake and the recognized word will 
be incorrect. If the random draw determines a 
correct response is to be given, the first word 
will be output to the system but in this case it 
is an error. This error was not expected; 
therefore, although the actual accuracy rate of 
the system will not be under <"he experimenters 
control the overall error rate should be very 
close to the percentage of times the runner up 
word is chosen. If a 100 % accura'^e system was 
used this percentage of runner up choices would be 
equal to the error rate observed. Since a 100 % 
accurate system does not exist it appears as if 
close will have to do. 

The Interstate Electronics Corporation machine is 
suitable for this type of task. The only problem involved 
with this recommendation is that it will be impossible to 
observe error rates much less than the underlying error rate 
associated with the equipment chosen for use in •^he 
experiment. 

Since there was evidence of the hypothesized negative 
correlation, it is possible that ■f'he frustration measurement 
will fulfill the needs of follow on experiments. This 
question of frustration measurement should be investiga'^ed 
further before undertaking the next phase of experimentation 
to answer the question about acceptable accuracy rates for 
voice r ecogni'*’ ion equipment. 
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APPENDIX A 



<-LIST 1.9000 



10 

20 

30 

40 

50 



= 

DIM 

DIM 



I 

F^(2) 



60 



>0001 4000 



” ■■ ; REM D$ CONTAINS A CNTRL D 
"5 REM It CONTAINS A CNTRL 
F$(3):Ft(l) = "COMPUTER" 
W$(200. 2) 

PRINT "DO YOU WANT TO:": PRTRT 

E A FILE" : PRINT 
INPUT A 

70 ON^ A GOTO T'000. 2000. 

1000 Bt = "CREATE" 

1010 GOSUB 6000 
1020' PRINT Dt;"OPEN 
1030 I = l:J = 1 

1040 GOSUB 5000 

GOTO 50 ' 

Bt = "UPDATE" 

GOSUB 6000 



= "STOCK" :FtC3) = "AIRLINE" 



TO'^'^ PR nTr^’’TT~CRERTE”A~l^EW LE" :“TRI NT 
PRINTOUT A FILE": PRINT "4. STOP" 



'2. UPDAT 



Ft ■( A )■ r'TV00 1 “ : ■ PRirm)tT " CLOSE'*- 



1050 

2000 

2010 



2020 

2030 

2040 

2050 

2060 

2070 



ONERR 
PRINT DS-. ' 
PRINT__D^; ' 

FOR i = r 

PRINT Dt; ' 
PRINT "DO 

'To ‘ 



V001 



2080 

2090 



2120 



GOTO 2060 
OPEN "? 

READ "i 
TO 200: 

CLOSE" 

YOU_WANT^ TO: " : PRINT "1. 
THE' LI ST OF W0RD6**-:^FmvlT 
AA 

GOTO 2100.2200.2300 
"WHAT WORD NUMBER DO 
INPUT A -1" 

l: IF I <0 THEN GOTO 2170 
PRINT "DO YOU 'WANT TO CHANGE" 

UND ALIKE" 



Ft(A) 

FtC A) 

“FQ-R"J ' = “rT0"2:'" INPUT WtTirj)':" NEXT-Tr:~NE)Cr-T- 



ADD 
INPUT 
ON AA 
2100 PRINT 
0 STOP 
2110 INPUT 



CHANGE INDIVIDUAL WORDS" 
-*''3rr“ST0P" 



PRINT " 



-RRIMT^ Z;~THEr-50 



2130 



2140 

2150 

2160 



INPUT J 
PRINT I.J' 
INPUT W$(L 
GOTO 2100 



J) 



2170 I = 201: GOSUB 
2180 GOTO 2070 
2200 PRINT "WHAT WORD 
I 



070 



NUMBER DO YOU WANT TO START AT?" 



2210 



2230 



INPUT I:J = 
GOSUB 5000 
GOTO 2070 
50 



GOTO 3060- 



2300 GOTO 
3000 Bt = "PRINTOUT" 
3010 GOSUB 6000 
3020 
3030 
_3040 

■3050 

3060 
3070 



D$; ' 
Dt; ■ 
= 1 
Dt; ' 
"DO 



OPEN ";Ft(A)! 
READ ":F$(;A) 

TO 200:- F0R2r- = 
CLOSE" 

YOU WANT": PRINT 



V001' 



0080 

3085 



ONERR 
PRINT 
PRINT 
FOR I 
PRINT 
PRINT 

1ST AND SOUND ALIKES'*^: PRINT^ 
INPUT J 

ON J GOTO 3090,3090.3400 



F TO 2"r^NPDT-TJta7-jr:“NEXT 3J: NEXT T “ i 
WORD LIST": PRINT "2. WORD u 



1 , 



JUST THE 
-STOP*’ 
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3(390 PRINT "DO YOU HAVE OKT^MAGE LQADED PND^ANT‘0 HARDCOPY? (Y OR N)“ 
3100 INPUT Yt 

3110 IF Y$ = "N" THEN GOTO 3170 

3120 PRINT‘D$;"PR<^ 2" 

3130 PRINT Ili, "S0N" 

3170 ON J GOTO 3200,3300,3400 



3200 


FOR T"^ 'T~TO 200: 


PRiNr~rrvji(Trm : 


"NEXr~I 


■ ■ 


3210 


PRINT Dt;"PR# 0": 


GOTO 3070 






3300 


FOR I = 1 TO 200: 


PRINT I,W$CI,J - 


l),W$a,J); NEXT I 




331 0‘ 


PRTnt D$; " PR#“0" 


GOTO“3070 ' 






3400 


GOTO 50 








4000 


STOP : END 








5000 


IF I ) 200 THEN 


GOTO“5^065 






5010 


PRINT I,J 








5020 


INPUT W$(I,J) 








5030 


IF W$a,J) = "END 


" THEN GOTO 5065 






5040 


J = J + 1 









5050 IF J = 2 THEN GOTO 5010 
50E0 I = I + i:J = l; GOTO 5000 
5065 GOSUB 5070: RETURN 
5070 PRINT D$; "OPEN " ; F$ ( A) ; " , V00 1 " 

5080’ PRINT D$; "WRITE ” ;F3i(.A) 

5090 FOR K = 1 TO I - i: FOR J = 1 TO 2: PRINT W^CK, J) : NEXT J: NEXT K 
5100 PRINT D$; "CLOSE": RETURN 

6000 PRINT "WHAT FILE DO YOU WANT TO ";B$: PRTRT^^. "COMPUTER" :“PRINT "2. 

STOCK": PRINT "3. AIRLINE" i 

6010 INPUT a: RETURN 




I 



I 
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Airline Vocabulary 



1 . 


A1 lentown 


Morgantown 


2. 


Abil ene 


Aberdeen 


3. 


Albuquerque 


Kodiak 


4. 


East Hampton 


Birmingham 


5. 


Washington Dulles 


Washington National 


6. 


Columbus Ohio 


Columbus, Georgia 


7. 


Akron 


Charleston 


8. 


Ohara Field 


Omaha 


9. 


Cleveland 


Lakeland 


10. 


Cedar Rapids 


Grand Rapids 


11 . 


Corpus Christi 


Cincinnati 


12. 


San Juan 


South Bend 


13. 


Boston 


Boise 


14. 


Rapid City 


Sioux City 


15. 


Daytona Beach 


Long Beach 


16. 


Goose Bay 


Pine Coy 


17. 


Dayton 


Galveston 


18. 


Utica 


Sitka 


19. 


Denver 


Beaver 


20. 


Lewiston 


Allentown 


21 . 


Detroi t 


Duluth 


22. 


Montpel ier 


Eau Claire 


23. 


Charleston 


Kinston 


24. 


Monterey 


San Jose 


25. 


Frankl in 


Richland 


26. 


Bangor 


Boston 


27. 


Fargo 


Elko 


28. 


Mani towoc 


Roanoke 


29. 


New York 


Norfol k 


30. 


San Jose 


San Diego 


31 . 


Fort Collins 


Ft Smith 


32. 


Laramie 


Miami 


33. 


Kona 


Helo 


34. 


San Diego 


San Francisco 


35. 


Freeport 


Bridgeport 


36. 


Missoula 


Honolul u 


37. 


Oklahoma City 


Jefferson City 


38. 


Colorado Springs 


Hot Springs 


39. 


Pi ttsburgh 


Plattsburgh 


40. 


Dul uth 


Detroit 


41 . 


Lousville 


Knoxville 


42. 


St Petersburg 


Rochford 


43. 


Honolulu 


Missoul a 


44. 


Ki rksvi 1 1 e 


Loui svil le 


45. 


Col umbia 


Atl anta 


46. 


Columbus Georgia 


Columbus, Ohio 


47. 


Roanoke 


Manitowoc 


48. 


Rockford 


Medford 


49. 


Washington National 


Washington Dulles 


50. 


Jacksonvil le 


Jackson 



30 



51 . 


Atlantic City 


Oklahmoa City 


52. 


Ch i CO 


Tol edo 


53. 


Amaha 


OHara Field 


54. 


PI attsburgh 


Pittsburgh 


55 . 


Eau Claire 


Montpel ier 


56. 


Norfol k 


New York 


57. 


Cheyenne 


San Juan 


58. 


San Jose 


Monterey 


59. 


Phel 1 ipsburgh 


St. Petersburg 


60. 


Hel 0 


Kona 


61. 


Medford 


New Bedford 


62. 


Beaver 


Denver 


63. 


Lew is town 


Lewiston 


64. 


Sitka 


Utica 


65. 


Galveston 


Dayton 


66. 


Grand Rapids 


Cedar Rapids 


67. 


Dubuque 


A1 buquerque 


68. 


Pine Coy 


Goose Bay 


69. 


Richland 


Cleveland 


70. 


Long Beach 


Daytona Beach 


71. 


South Bend 


Cheyenne 


72. 


Birmingham 


East Hampton 


73. 


Sioux City 


Rapid City 


74. 


Ci ncinnati 


Corpus Christi 


75. 


Kenston 


Akron 


76. 


Acapulco 


Chico 


77. 


Nome 


Rome 


78. 


Orl ando 


Kokoma 


79. 


Pompano Beach 


West Palm Beach 


80. 


Harrisburg 


Paris 


81 . 


Hot Springs 


Colorado Springs 


82. 


Pittsfield 


Spri ngfield 


83. 


Kodiak 


Dubuque 


84. 


Milwaukee 


Menominee 


85. 


Knoxvil 1 


Kirksville 


86. 


Ft. Smith 


Fort Col 1 ins 


87. 


Paris 


Harrisburg 


88. 


Morgantown 


Lewi s town 


89. 


Springfield 


Pittsfield 


90. 


Elko 


Fargo 


91 . 


New Bedford 


Phi 1 1 i psburg 


92. 


West Palm Beach 


Pompana Beach 


93. 


Bridgeport 


Freeport 


94. 


Kokomo 


Orl ando 


95. 


Aberdeen 


Abilene 


96. 


Jackson 


Jacksonvi 1 1 e 


97. 


Menomimee 


Milwaukee 


98. 


Fort Wayne 


Fort Lauderdale 


99. 


Rome 


Nome 


100. 


Toledo 


Acopulco 


101 . 


Rochester 


Worchester 


102. 


Kal amozoo 


Waterloo 


103. 


Providence 


Provincetown 
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104. 


Morristown 


Worthington 


105. 


Cl arksburg 


Marthas Vineyard 


106. 


Tal lahassee 


Apple Valley 


107. 


San Angelo 


Show Low 


108. 


Martha Vineyard 


Ogdensburg 


109. 


Gai nesvi 1 1 e 


Knoxvi 1 1 e 


no. 


Sacremento 


Larado 


Ill . 


Lake Charles 


Lake Tahoe 


112. 


Osage Beach 


Anchorage 


113. 


Brookings 


New Orleans 


114. 


Pueblo 


San Angelo 


115. 


Anti guo 


A1 toona 


116. 


Knoxvi 1 1 e 


Watervi 1 1 e 


117. 


Larado 


El Dorado 


118. 


Ogdensburg 


Lynchbur g 


119. 


Sumter 


Rochester 


120. 


Show Low 


Tampi co 


121 . 


Wichita Falls 


Wichita 


122. 


Worthington 


Riverton 


123. 


Provincetown 


Providence 


124. 


Waterloo 


Kalamazoo 


125. 


Apple Valley 


Tallahassee 


126. 


Appl eton 


Trenton 


127. 


Modesto 


Durango 


128. 


Worchester 


Sumter 


129. 


Huntsvil le 


Catski 1 1 s 


130. 


Watervil le 


Janesville 


131 . 


Baton Rouge 


Osage Beach 


132. 


Marquette 


Phoenix 


133. 


New Orleans 


Bi 1 1 i ngs 


134. 


Walla Walla 


Antingua 


135. 


Tupelo 


Modesto 


136. 


Astoria 


Augusta 


137. 


Catskil 1 s 


Greinvil le 


138. 


Atlanta 


Bermuda 


139. 


Janesville 


Huntsville 


140. 


Durango 


Buffal 0 


141 , 


Newburg 


Cl arksburg 


142. 


Trenton 


Grand Junction 


143. 


Bill ings 


Brooki ngs 


144. 


Fayettevil le 


Gainsesville 


145. 


Greenvil 1 


Evansvil le 


146. 


Buffalo 


Greensboro 


147. 


Evansvil le 


Jacksonville 


148. 


Phoenix 


Marquette 


149. 


Greensboro 


Ontario 


150. 


Grand Junction 


Morristown 


151 . 


Anchorage 


Baton Rouge 


152. 


Wichita Falls 


Twin Falls 


153. 


San Francisco 


San Juan 


154. 


Watervil le 


Greenvil 1 e 


155. 


San Juan 


San Jose 


156. 


Princeton 


Pendleton 
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157. 


Churchill Falls 


Idaho Falls 


158. 


Hastings 


Harrison 


159. 


Wichita 


Wichita Falls 


160. 


Fort Lauderdale 


Fort Wayne 


161. 


Puebl 0 


Sacramento 


162. 


Saint Louis 


St. Croix 


163. 


El Dorado 


Pueblo 


164. 


Grand Cayman 


Grand Canyon 


165. 


Jefferson City 


Atlantic City 


166. 


Idaho Falls 


Churchill Falls 


167. 


Pendl eton 


Pri nceton 


168. 


Greenville 


Waterville 


169. 


Miami 


Laramie 


170. 


Grand Canyon 


Grand Cayman 


171 . 


St Croix 


Saint Louis 


172. 


Lakeland 


Frankl i n 


173. 


Harrison 


Hasti ngs 


174. 


Boise 


Bangor 


175. 


Twin Falls 


Wichita Falls 


176. 


Riverton 


Cl inton 


177. 


Ontario 


Orlando 


178. 


Lynchburg 


Galesburg 


179. 


A1 toona 


Astoria 


180. 


Agusta 


Wausau 


181 . 


Victoria 


Topeka 


182. 


Sarosota 


Pensacola 


183. 


Bermuda 


Col umbi a 


184. 


Poplar Bluff 


Council Bluffs 


185. 


Tul sa 


Guadalahara 


186. 


Jacksonville 


Danville 


187. 


Poza Rica 


Walla Walla 


188. 


Guadalahara 


Sarasota 


189. 


Pensacola 


Poza Rica 


190. 


Orl ando 


Mankato 


191 . 


Topeka 


Til Isa 


192. 


Wausau 


Victoria 


193. 


Galesburg 


Newburg 


194. 


Penta Gorda 


Tupel 0 


195. 


Lake Tahoe 


Lake Charles 


196. 


Danville 


Fayettevil le 


197. 


Mankato 


Pueblo 


198. 


Hattiesburg 


Clarksburg 


199. 


Lampico 


Punta Gorda 


200. 


Cl inton 


Appleton 
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1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

n 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 



Computer vocabulary 



Login 

A1 ter 

Backward 

Command 

Cursor Column 

Findup 

Left 

Nfind 

Put-D 

Query 

Reset 

Set Autosave 

Set File Mode 

Set Line Character Off 

Set Number 

Set Reserved 

Set Synonym 

Set Verify 

Stack 

Delete Line 

Assign 

Five 

Six 

Seven 

Eight 

Ni ne 

Ten 

A1 pha 

Bravo 

Chari ie 

Logoff 

Bottom 

Cmsg 

C-Repal ce 
Find 

Join Cursor 

Msg 

Next 

Purge 

Repl ace 

Set Arbitrary Character 
Set Filler 

Set Line Character On 

Set Nulls 

Set Record Format 

Set Stream 

Set Variable Blank 

Spl it 

Uppercase 

Duplicate Line 

Access 

Filedef 

Global 

Listfile 

Start 



Logoff 

Add 

Bottom 
Compress 
Cursor File 
Forward 
Load 

Nfind-up 

Parse 

Quit 

Restore 

Set Case 

Set File Name 

Set Logical Record Length 

Set Pack 

Set Scale 

Set Table! ne 

Set Wrap 

Status 

Duplicate Line 

Access 

Four 

Sixty 

Seventy 

Eighty 

Ninety 

Send 

Papa 

Romeo 

Wh i s key 

Logi n 

Backward 

Command 

Cursor Column 

Findup 

Join 

Macro 

Nfind 

Put-D 

Reset 

Set Autosave 

Set File Mode 

Set Line Character Qff 

Set Number 

Set Reserved 

Set Synonym 

Set Verify 

Stack 

Up 

Delete Line 

As s i gn 

Fetch 

Cobol 

Fil edef 

State 






56 

57 

58 

59 

60 

61 

62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 

74 

75 

76 

77 

78 

79 

80 

81 

82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

94 

95 

96 

97 

98 

99 

100 

101 

102 

103 

104 

105 

106 

107 

108 

109 

no 



Zero 

One 

Two 

Three 

Four 

Profile 

Cancel 

Cms 

Cp 

Expand 

File 

Join Column 
Modi fy 
Power- Input 
Renumber 

Selective Change 

Set Current Line 

Set Image 

Set Message Mode 

Set Prefix 

Set Span 

Set Text 

Sort 

Type 

Debug 

Papa 

Quebec 

Romeo 

Sierra 

Tango 

Uni form 

Victor 

Wh i s key 

Yankee 

Zulu 

No Profile 

C-Append 

C-Locate 

C-overlay 

Dupl i cate 

Emsg 

Put 

Join 

Move 

Preserve 

Repeat 

Set APL 

Set Escape 

Set Implicitly Cms 

Set Nondisplayable Characters 

Set Range 

Set Stay 

Set Truncate Column 

Sos 

Up 



Kilo 

Echo 

Echo 

Thirty 

Five 

No Profile 

C-Append 

Cmsg 

C-Replace 

Emsg 

Find 

Join Cursor 
Move 

Preserve 
Repeat 
Set APL 
Set Escape 

Set Implicityly to CMS 
Set Nondisplayable Characters 
Set Range 
Set Stay 

Set Truncate Column 

Sos 

Up 

Disk 

A1 pha 

Stack 

Tango 

A1 pha 

Romeo 

Nine 

Preserve 

X-ray 

Chari ie 

Move 

Profile 

Cancel 

Cms 

Cp 

Down 

Expand 

Input 

Join Column 
Msg 
Purge 
Replace 

Set Arbitrary Character 

Seet Filler 

Set Line Character On 

Set Nulls 

Set Record Format 

Set Stream 

Set Variable Blank 

Spl i t 

Uppercase 
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Ill 

112 

113 

114 

115 

116 

117 

118 

119 

120 

121 

122 

123 

124 

125 

126 

127 

128 

129 

130 

131 

132 

133 

134 

135 

136 

137 

138 

139 

140 

141 

142 

143 

144 

145 

146 

147 

148 

149 

150 

151 

152 

153 

154 

155 

156 

157 

158 

159 

160 

161 

162 

163 

164 

165 



Disk 

X-Ray 

Twenty 

Thi rty 

Forty 

Fi f ty 

Sixty 

Seventy 

Eighty 

Ninety 

Xedi t 

C-Delete 

C-Last 

Compress 

Cursor File 

Forward 

Get 

Load 

Nfi nd-up 

Overlay 

Qui t 

Restore 

Set Case 

Set File Name 

Set Logical Word Length 

Set Pack 

Set Scale 

Set Tabl i ne 

Set Wrap 

Status 

Top 

Compare 

Erase 

Fetch 

State 

Del ta 

Echo 

Foxtrot 

Gulf 

Hotel 

Update 

C- First 

C-Insert 

Copy 

Cursor Screen 

Delete 

Set 

Hel p 

Locate 

On 

Q-Qui t 
Read 
Ri ght 

Set Command Line 
Set File Type 



Debug 

Yankee 

Ninety 

Eighty 

Seventy 

Five 

Fi f ty 

Seven 

Thirty 

Twenty 

Update 

C-Fi rst 

C-Locate 

Copy 

Cursor Screen 

Find-up 

Set 

Locate 

Next 

On 

Q- Qu i t 
Ri ght 

Set Command Line 
Set File Type 
Set Macro 

Set Program Function Key 

Set Screen 

Set Tabls 

Set Zone 

Stack 

Transfer 

Cp 

Di sk 
Filedef 
Start 
Indi a 
One 

Fi ndup 

Off 

Hel p 

Xedit 

Change 

C-Last 

Count 

Cursor File 

Down 

Get 

Hextype 

Lowercase 

Off 

Query 

Recover 

Read 

Set Column Pointer 
Set Hexidecimal 






166 

167 

168 

169 

170 

171 

172 

173 

174 

175 

176 

177 

178 

179 

180 

181 

182 

183 

184 

185 

186 

187 

188 

189 

190 

191 

192 

193 

194 

195 

196 

197 

198 

199 

200 



Set Macro 

Set Program Function Key 

Set Screen 

Set Tabis 

Set Zone 

Transfer 

CP 

Cobol 
India 
Jul iett 
Kilo 
Lima 
Mi ke 

November 

Oscar 

Add 

Change 

Count 

Down 

Hextype 

Input 

Lowercase 

Macro 

Off 

Parse 

Recover 

Save 

Set Column Pointer 
Set Hexidecimal 
Set Mask 
Set Point 
Set Serial 
Set Terminal 
Set Equals 
Shi ft 


Set Mask 

Set Point 

Set Serial 

Set Terminal 

Set Equals 

Type 

Compare 

Global 

Lima 

Delete 

Zero 

India 

Quit 

Oscar 

A1 ter 

A1 ter 

C-Insert 

C-Overlay 

Dupl icate 

Hel p 

Put 

Left 

Modi fy 

Overlay 

Power-Input 

Renumber 

Selective Change 
Set Current Line 
Set Image 
Set Message Mode 
Set Prefix 
Set Span 
Set Text 
Set Prefix 
Sort 
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Stock vocabulary 



1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

49 

50 

51 

52 

53 

54 

55 



Houston Natural Gas 

IBM 

ITT 

Iowa Electric 
K-Mart 

Kaiser Aluminum 
Lenox 

Leer-Siegl er 
Lionel 
Iowa Gas 
Litton 

Kaiser Steel 

Lockheed 

MGM 

Magic Chef 

Marriott 

Marathan Oil 

Mattel 

Maytag 

Memorex 

McDonalds 

AMF 

McDonald-Douglas 

3M 

NCR 

Nabisco 
Natomas 
Oklahoma Gas 
Penney' s 

Combustion Engineering 

Can Edeson 

Dennys 

Macys 

Dow Chemical 

Elgin 

Dorsey 

Florida Power & Light 
Greyhound 
Hal i burton 
Kroehl er 

Wisconsin Power & Light 

Grumman 

U.S. Air 

U.S. Steel 

U.S. Life 

Western Union 

Zapata 

Air Florida 

Wennebago 

Air Wisconsin 

Atlantic Richfield 

BDM 

Tampox 

Alabama Power 



Oklahoma Gas 

MGM 

IBM 

Iowa Gas 
Gill ette 
Kaiser Steel 
Purex 
Si nger 
Mattel 

Iowa Electric 
Hal iburton 
Kaiser Aluminum 
Square D 
IBM 

Zen i th 
Mattel 
Ashland Oil 
Marriott 
Rubber Maid 
Tampox 

McDonal d- Dougl as 
ATT 

McDonal ds 

BDM 

CBS 

Sambos 

Ameroda Hess 
Houston Natural Gas 
Macy's 

Cummins Engine 

Western Union 

Dorsey 

Penneys 

Dow Jones 

Hal iburton 

Dennys 

Wisconsin Power & Light 

Grumman 

Elgin 

Burroughs 

Florida Power & Light 

Greyhound 

U.S. Gypsum 

U.S. Air 

U.S. Steel 

Con Edison 

Aenith 

Air Wisconsin 

Whi rl pool 

Air Florida 

Atlantic Research 

MGM 

Zerox 

Florida Power & Light 
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56 

57 

58 

59 

60 

61 

62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 

74 

75 

76 

77 

78 

79 

80 

81 

82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

94 

95 

96 

97 

98 

99 

100 

101 

102 

103 

104 

105 

106 

107 

108 

109 

no 

111 



Bank of Virginia 

Burroughs 

CBS 

Atlantic Research 

Zenith 

Xerox 

Bank of America 

John Deer 

Delta Airlines 

Dow Jones 

Eastern Airlines 

General Motors 

General Steel 

General Radio 

Georgia Pacific 

Goodyear 

Gillette 

Gulf Oil 

Hicla Mining 

Goodrich 

Heinz 

Pei rex 

Sambos 

Singer 

Square D 

Collins Radio 

Tandy 

Texaco 

United Airlines 
U.S. Gypsum 
Ala Moana 
A1 bertsons 
Gul f Western 
Dome Mining 
Ameroda Hess 
American Broadcasting 
Honda 

Allis Chalmers 
American Hospital 
Ashland Oil 
Bank of America 
Bendfx 

Boise Cascade 

Canadian Pacific 

Ben Gay 

Champion 

Chrysler 

Coca Cola 

Clorex 

Chase Manhattan 
Data General 
Cities Services 
Canoca 

Southern Pacific 

Boeing 

Ford 



Bank of America 

Kroehl er 

NCR 

Atlantic Richfield 

Zapato 

Lenox 

Bank of Virginia 
Chrysler 

Eastern Airlines 
Dow Chemical 
Delta Airlines 
General Radio 
Data General 
General Motors 
Canadian Pacific 
Goodrich 
K-Mart 

Gulf Western 
Dome Mining 
Goodyear 
Honda 
Cl orox 

Collins Radio 
Leer Siegler 
Tandy 
Texaco 
Square D 
Nabisco 

Eastern Airlines 
U.S. Life 
Coca Cola 
Alles Chalmers 
Gulf Oil 
Hi do Mining 
Natomas 

American Hospital 
Heinz 

A1 bertsons 

American Broadcasting 
Marathan Oil 
Bank of Cal i fornia 
Ben Gay 
Ben Gay 

Georgia Pacific 
Bendex 

Chose Manhatten 
John Deer 
Ala Moana 
Meraorex 
Champion 
General Steel 
Citicorp 
Coca Cola 
Georgia Pacific 
Boise Cascade 
Fotomat 
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112 

113 

114 

115 

116 

117 

118 

119 

120 

121 

122 

123 

124 

125 

126 

127 

128 

129 

130 

131 

132 

133 

134 

135 

136 

137 

138 

139 

140 

141 

142 

143 

144 

145 

146 

147 

148 

149 

150 

151 

152 

153 

154 

155 

156 

157 

158 

159 

160 

161 

162 

163 

164 

165 



El Paso Gas 

Union Oil of California 
TRW 

Ci ticorp 
Revlon 
Woolworth 
Rockwell 

United Technologies 

Fotomat 

San Diego Gas 

Southwestern Gas 

Bell & Howell 

Honeywel 1 

Avon 

Threshold Technology 
Allied Corp. 

ACF Industries 
Beckman Instruments 
Big Three 
Boston Edison 
Brooklyn Union Gas 
Bui ova 

Campbell Soup 
Carolina Power & Light 
Carpenter Technology 
AMF 

Becton Dickinson 
Carolina Freight Carriers 
Central Maine Power 
Clark Oil & Refining 
Col dwell Banker 
Colonial Store 
Commonwealth Edison 
Central & South West 
Consolidated Foods 
Continental Can 
Colt Industries 
Clark Equipment 
Central Soya 
Continental Oil 
Copper Range 
Consolidated Freightway 
Continental Air Lines 
Corning 

Cummins Engine 
Cooper Laboratories 
Cyprus Mines 
Del Monte Corp 
Continental Telephone 
Detroit Edison 
Diebold 

Dresser Industries 
Eastman Kodak 
Crocker 



San Diego Gas 
Bank of California 
NCR 

Cities Services 
Avon 

Whi rl pool 
Honeywell 

Threshold Technology 
Ford 

El Paso Gas 
San Diego Gas 
Rockwel 1 
Rockwel 1 
Revlon 

United Technology 
Alles Chalmers 
AMF 

Becton, Dickinson 
Square D 
Con Edison 
San Diego Gas 
Burroughs 
Honeywel 1 

Wisconsin Power & Light 
United Technology 
ACF Industries 
Beckman Instruments 
Carolina Power & Light 
Central & South West 
Clark Equipment 
Rockwel 1 
Colt Industries 
Con Edison 
Central Soya 
Consolidated Freightway 
Continental Air Lines 
Colonial Stores 
Clark Oil & Refining 
Central & South West 
Continental Telephone 
Cooper Laboratories 
Consolidated Foods 
Continental Can 
Crocker 

Combustion Engineering 
Cooper Range 
Bucyrus-Erie 
Delta Air Lines 
Continental Oil 
Commonwealth Edison 
Di Giorgio 
Dreyfess 

Eastern Airlines 
Corning 
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166 

167 

168 

169 

170 

171 

172 

173 

174 

175 

176 

177 

178 

179 

180 

181 

182 

183 

184 

185 

186 

187 

188 

189 

190 

191 

192 

193 

194 

195 

196 

197 

198 

199 

200 



Bucyrus-Erie 

Emuson Electric 

Empire Gas 

Di Georgia 

Dreyfus 

Faberge 

Federal Mogul 

Fieldcrest 

Fairchild 

Emery Air Freight 

Ford Motor 

General Dynamics 

Gerber 

Grand Union 

Hushey 

Mil ton 

Host 

Inland Steel 
Internation Harvester 
Johnson & Johnson 
Kennecaft Copper 
Lone Star Gas 
Lukens Steel 
First Chicago 
General Food 
Holiday Inn 
Fo remost 
Hewlett Packard 
Houston Natural Gas 
International Paper 
Johnathan Logan 
Magnavox 

Lone Star Industries 
Federal Paper Board 
Kimberly Clark 



Cyprus Mines 
Emery Air Freight 
Emery Air Freight 
Georgia Pacific 
Dresser Industries 
Fairchild 

Federal Paper Board 
First Chicago 
Faberge 

Emerson Electric 
Fo remost 
General Food 
Gillette 
Estern Union 
Hewlett- Packard 
Holiday Inn 
Houston Natural Gas 
Kaiser Steel 
International Paper 
Jonathan Logan 
Kimberly Clark 
Lone Star Industries 
Kaiser Steel 
Fieldcrest 
General Dynamics 
Hil ton 
Ford Motor 
Hershey 
Host 

International Harverster 
Johnson & Johnson 
Lenox 

Lone Star Gas 
Federal Mogul 
Kennecott Copper 
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APPENDIX C 






<-LILI ST 1 ,,9000 

10 DIM W$(200, 2) , F$<;3) , P( 4) 

20 D$ = REM CONTRINS ft CONTROL-D 

30'FiCD = "COMPUTER" :F^C2) = "STOCK" :'F$T3) = "OrRUTNE:^ 

40 P<1) = .99:P(2) = .35:P(3) = .90:P(4) = .85 
45 = " " _ 

50 PRINT ‘"WHAT WORD DflTOBflSE DO YOU WftNT TD~USE?": PRINT "^1. XOMPUTER": 
"2. STOCK": PRINT "3. AIRLINE" 

60 I NPUT A ^ 

70 PRINT D$;"OPEN " ; F$( A) ; " , V0O1 " 

60' PRINT D^;"READ ";F^(A) 

90 FOR I = 1 TO 200: FOR J = 1 TO 2: INPUT W$(I,J): NEXT J: NEXT I 

T00“ PR I NT“D3i“; " clTose ■ 

105 PRINT "INPUT A SEED": INPUT IS 
110 PRINT "WHAT ALGORITHM DO YOU WANT TO USE?" 

120“ INPUT B: IF B <“ 0 THEN STOP ~ 

130 GOSUB 40000 

140 CALL JTABLE + 15 

150 VOC$ = "CLASS'IC. VOC" ^ ^ ; 

160 GOSUB 40100 

170 PRINT "FIRST SOME PRACTICE WORDS" 

1S0 I = i 

190 PRINT I,WSCI,1): GOSUB 40400: FOR Z = 1 TO 150: NEXT Z: GOSUB 300 
200 IF I <11 THEN GOTO 190 
g"!!) Q = 0:“S = 0:N = 0: T = "0 

220 PRINT "WE WILL NOW START THE EXPERIMENT" 

230 1=1 

240 “ PRINT I,WS(I,1): GOSUB 40400TTTJR Z = 1 1U 150: NhX 1 z: T3O5UET-300 
250 IF I < 201 THEN GOTO 240 

260 PRINT "THANK YOU FOR PARTICIPATING IN THE EXPERIMENT" 

I 27 0““t i: 200 - - N ) / . 2 ! 

280 PRINT B;T; ; Q; ; s ; ;n . i 

230 GOTO 110 i 

300 IF RND (IS) > P(B) THEN GOTO“320 
310 PRINT M$;W$(I,1): GOTO 400 
320 R = RND (IS) 

330 IF R < .33 THEN GOTF~3“90 

340 IF R < .66 THEN GOTO 3B0 

350 PRINT M$;"?":Q = Q + 1: GOTO 400 

360 U'~^ INT (~R“ND ( I ST”ST-2'00)’^+T 

370 IF U = I THEN GOTO 360 

380 PRINT M$;W$(U, 1):N = N + 1: GOTO 400 

390 PRINT M3>; W$Tl, 2“) :S~= S^ 

400 FOR Z = 1 TO 150: NEXT Z:I = I + 1-* PRINT : RETURN 



Appendix D 



Code # 



CATEGORY [ ] AIRLINE 

[ ] STOCK BROKER 
[ ] COMPUTER 

TIME SPENT AT A COMPUTER TERMINAL A DAY 



Age 



hrs 



1. Have you used voice recognition equipment before 

2. Have you seen voice recognition equipment used before 



□ 


YES 


□ 


NO 


□ 


YES 


o 


NO 


make, 


and 



considering the number of errors you saw the voice system make, how well 
do you like the voice system. 



Really 


Don ' t 


Neutral ; 


Li ke 


Really 


Dislike 


Like 


If I have it, fine 

If I don't have it, fine 


It 


Like It 


□ 


□ 


□ 


□ 


□ 



Is the accuracy of the voice recognition system adequate enough to make you 
want to use it in your daily job? 



Very 

Inadequate 


Slightly 

Inadequate 


Neutral 


SI i ghtly 
Adequate 


Very Adequa 


□ 


□ 


□ 


□ 


□ 


Comparing voice input to manual typing 


input, is voice 


input: 


Undoubtedly 


Moderately 


The 


Modera tely 


Undoubtedly 


Worse 


Worse 


Same 


Better 


Better 


□ 


□ 


o 


□ 


n 
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6. Do you consider yourself a slow, intermediate, or fast typist? 

Slow Intermediate Fast 

□ □ □ 

7. How accurately do you think you type? 

Poor Fair Quite Well 

ana 

8. Did you ever take a typing course? 

Yes No 

a a 

9. What part of the country, (USA) did you grow up in? 

10. How many years experience have you had in typing information into computers? 
years . 

11. Please tell the experimenter how much and what type of educational background 
you have. 
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2 

22 

27 

17 

21 

21 

20 

16 

22 

18 

14 

15 

18 

16 

15 

20 

15 

20 

15 

16 

15 

12 

13 

17 

13 

13 

13 

12 

12 

11 

12 

i;< 

; ( 

8 

12 

11 

12 

10 

10 

10 

10 

6 

9 

9 

8 

8 

7 

7 

7 

7 

7 

7 

7 

7 

3 

5 

5 

3 

2 

3 

2 









APPENDIX E 


Paw 


Dat 3 










3 


4 


5 


6 


7 


8 


9 


10 • 


1 1 


12 


13 


1 4 


15 


16 


17 


18 


20 


10 


12 


35 


1 


1 


1 


4 


5 


4 


13 


2 


2 


1 


1 


1 


1 1 


14 


13 


27 


2 


2 


2 


4 


4 


4 


12 


2 


2 


1 


2 


1 


19 


12 


5 


32 


1 


2 


2 


3 


2 


2 


7 


2 


3 


2 


1 


1 4 


1 1 


10 


11 


27 


3 


2 


2 


4 


2 


5 


1 1 


1 


2 


1 


2 


1 1 


1 1 


15 


6 


28 


3 


2 


1 


4 


4 


4 


12 


1 


2 


1 


3 


1 


1 1 


9 


1 1 


38 


6 


2 


2 


5 


5 


5 


15 


2 


2 


1 


1 


1 


1 4 


9 


7 


34 


1 


1 


2 


4 


4 


4 


12 


2 


2 


1 


1 


0 


7 


9 


13 


28 


2 


2 


1 


4 


5 


5 


14 


3 


3 


1 


1 


1 1 


1 1 


12 


6 


38 


2 


2 


2 


5 


4 


5 


14 


1 


3 


2 


4 


1 0 


15 


7 


7 


33 


1 


2 


1 


5 


5 


5 


15 


1 


1 


2 


3 


5 


1 4 


Q 


6 


27 


2 


2 


1 


5 


4 


5 


14 


2 


3 


2 


1 


4 


1 1 


1 2 


6 


3 1 


5 


2 


1 


2 


1 


2 


5 


3 


3 


1 


2 


8 


12 


8 


8 


33 


2 


2 




3 


5 


5 


13 


3 


3 


1 


3 


5 


13 


5 


10 


32 


1 


1 


1 


4 


4 


4 


12 


2 


2 


1 


3 


0 


6 


6 


14 


29 


2 


1 


1 


4 


4 


3 


1 1 


2 


2 


1 


3 


1 


9 


9 


6 


27 


2 


2 


2 


3 


2 


2 


7 


1 


3 


2 


3 


9 


4 


7 


13 


29 


2 


1 


1 


4 


2 


4 


10 


2 


2 


1 


2 


1 0 


8 


6 


9 


3 1 


2 


2 


2 


3 


4 


3 


10 


1 


2 


o 


2 


0 


7 


4 


12 


28 


5 


1 


1 


2 


2 


1 


5 


2 


3 


1 


9 


1 2 


7 


9 


6 


33 


1 


1 


1 


4 


5 


5 


14 


1 


1 


2 


2 


0 


1 0 


7 


5 


30 


1 


2 


2 


5 


5 


2 


12 


3 


3 


1 


3 


2 


8 


9 


4 


32 


6 


2 


2 


2 


3 


2 


7 


7 


3 


1 


3 


6 


4 


4 


1 3 


34 


1 


2 


1 


5 


5 


5 


15 


2 


2 


2 


1 


2 


7 


3 


<0 


36 


1 


2 


2 


3 


2 


2 


15 


1 


3 


2 


1 


1 


7 


3 


10 


26 


1 


2 


2 


3 


4 


5 


12 


2 


2 


1 


3 


0 


7 


6 


7 


38 


3 


2 


1 


2 


2 


4 


8 


2 


3 


1 


2 


1 4 


7 


7 


5 


29 


0 


2 


2 


4 


4 


4 


12 


2 


2 


1 


5 


0 


6 


5 


7 


26 


1 


2 


2 


5 


5 


5 


15 


2 


2 


1 


2 


3 


6 


5 


6 


3 5 


0 


2 


1 


4 


4 


4 


12 


1 


2 


1 


3 


6 


5 


3 


9 


35 


3 


1 


1 


4 


5 


5 


14 


3 


3 


2 


3 


10 


3 


5 


q 


30 


1 


o 


1 


4 


5 


4 


13 


2 


2 


1 


1 


2 


6 


5 


6 


27 


0 


2 


1 


5 


5 


5 


15 


2 


2 


2 


3 


1 


8 


6 


2 


39 


2 


2 


2 


4 


5 


4 


13 


1 


2 


1 


6 


0 


3 


8 


4 


29 


3 


2 


2 


4 


5 


4 


1 3 


3 


3 


1 


2 


3 


4 


6 


5 


35 


0 


2 


1 


4 


4 


4 


12 


2 


1 


2 


1 


1 


3 


6 


6 


3 1 


2 


2 


1 


5 


5 


5 


15 


2 


2 


1 


3 


0 


4 


8 


2 


33 


2 


2 


2 


3 


5 


4 


12 


3 


3 


1 


2 


3 


3 


6 


4 


35 


1 


1 


1 


4 


5 


4 


13 


2 


2 


2 


1 


2 


3 


6 


4 


32 


0 


2 


1 


3 


4 


4 


1 1 


1 


1 


1 


5 


3 


3 


6 


4 


37 


2 


2 


1 


5 


5 


5 


15 


2 


2 


1 


5 


9 


6 


2 


4 


26 


2 


2 


1 


4 


4 


4 


12 


2 


3 


1 


3 


4 


3 


5 


r, 
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